1. Field of the Invention
The invention relates to a graphical user interface for accessing information stored in a computer. More particularly, the invention relates to a user definable graphical interface for a computer operating system which utilizes pictorial information and animation as well as sound.
2. State of the Art
Very early computers were provided with a minimal interface which often consisted of little more than switches and lights. Rows of switches were set in positions representing binary numbers to provide input and rows of lights were illuminated representing binary numbers to provide output. Eventually, computer input and output included text and decimal numbers, which were input using punch cards and output using line printers. A major advance in computing was the interactive video display terminal (VDT). Early VDTs displayed several lines of alphanumeric characters and received input from a "QWERTY" keyboard. VDTs were a great improvement over switches and lights and even over punch cards and line printers.
As computers became more complex, it became necessary to systematize the manner in which information was stored and retrieved. The hierarchical file system was developed and is still substantially the only system in use today with a few exceptions. Under the hierarchical file system, information is stored in files and files are stored in directories. Directories may be stored in other directories and called sub-directories. Using this system, any file can be located by using a path name which identifies the path from a root directory through one or more subdirectories to the file; e.g., a typical path name may take the form: "rootdirectory/directory/subdirectory/filename".
In addition to the development of the hierarchical file system was the development of various "operating systems". The early computers did not require an "operating system" per se. They were manually programmed to perform a single task and then reprogrammed for a different task. Programs were stored on punch cards or tape and were loaded directly into the computer's random access memory (RAM) individually when needed by a system operator. With the development of various file systems, including the hierarchical file system, various programs and data could be stored on the same medium and selected for loading into the computer's random access memory (RAM). An operating system is a program which is used to access information on a storage medium and load it into RAM. The operating system allows the computer user to display the contents of directories and choose programs to be run and data to be manipulated from the contents of the directories. Every operating system, therefore, has a user interface, i.e. a manner of accepting input from a user and a manner of displaying output to a user. The input typically includes commands to the operating system to find information in directories, to display the contents of directories, to select files in directories for execution by the computer, etc. In addition, operating systems provide means for the user to operate on files by moving them, deleting them, copying them, etc. Output from the operating system typically includes displays of the contents of directories, displays of the contents of files, error messages when a command cannot be completed, confirmation messages when a command has been completed, etc. With many operating systems, when a program is selected for execution through the operating system, the selected program takes over control of the computer and returns control to the operating system when the program is ended. Modern operating systems share control with programs and several programs can run while the operating system is running.
The most primitive operating system interface is known as a "command line interface". While this type of interface is not necessarily indicative of a primitive operating system, it is primitive as an interface. The command line interface is purely text and presents the user with an arbitrary "prompt" such as "C:.backslash." or "%.backslash.:". The only information conveyed to the user by the command line prompt is that the operating system is ready to receive a command, and in the case of "C:.backslash.", that the operating system will perform commands with reference to the currently selected root directory "C". The commands to which the operating system will respond are most often obscure abbreviations like DIR to display the contents of the currently selected directory and CD to select a different directory. Moreover, the responses provided by the operating system interface to commands such as DIR may be equally obscure such as displaying a rapidly scrolling list of directory contents or the cryptic "Abort, Retry, Fail" message. Thus, in order to explore the contents of a file system using a command line interface, the user must repeatedly type DIR and CD and try to remember how the scrolling lists of filenames relate to each other in the hierarchy of the file system. Most users find this to be a tedious and trying experience.
More recently, the command line interface has been abandoned in favor of a fully graphical user interface ("GUI") such as those provided by the Apple Macintosh operating system and the IBM OS/2 operating system. To date, GUI interfaces to the operating system have been "WIMP" interfaces; that is they use Windows, Icons, Menus, and Pointers. In the development of WIMP interfaces, a central issue has been the organization of information for display on a the limited viewspace provided by a computer monitor. This issue has bee?r addressed by using the metaphor of a messy desktop to guide the design and layout of information on the graphical display. The metaphor of a messy desktop, which arose in the research on Rooms, and more recently 3-D Rooms, has become universal as an organizing paradigm for the display of user interactions with a computer operating system. In addition to the Macintosh and OS/2 operating systems interfaces, Unix systems X-windows, Microsoft Windows, and others are based on this metaphor. In a WIMP interface, windows are used to demarcate regions of the display assigned to individual programs, graphical icons are used to represent objects such as files and directories known to the operating system, menus can be displayed to list text string names of available operations, and a pointing cursor is used to select object icons or menu items that are visible on the display.
Graphical layouts provided by movable windows,icons, and menus of the WIMP interface have been very successful in helping to organize information, particularly data from alternative programs in progress, on a computer display. Nevertheless, they are offer limited functionality for depiction of operating system procedures and for graphical information about the files and directories present in the file system. Most computer users find the graphical interface to be much easier to learn and much easier to use than the command line interface. Many people have described the graphical interface as "intuitive". However, some people do not find it so intuitive and need more time to learn how to use it than do others.
Despite their vastly enhanced use compared to command line interfaces, the graphical interfaces presently used for access to operating system functionality are still somewhat regimented. For example, the icons are typically all rectangular and of the same size, e.g. 32 by 32 pixels. They are also generally generic. That is to say, for example, that a document concerning the subject of elephants would have the same icon as a document concerning the subject of roses. Typically, all of the directory icons are identical graphics of a folder with no graphical indication of what the folders contain either in subject matter or in amount of information. A folder containing one file has the same size and shape icon as a folder containing twenty files. Thus file and folder icons must always be accompanied by a text string for identification. Moreover, all the windows drawn in the GUI are identical except for the text string which typically appears in the window's title bar. Thus there is no graphical information presented to inform the user what directory is being viewed. The user must read the title bar and remember the association between the text string and the directory contents to determine what directory is being viewed.
There have been a number of extensions to the early WIMP interfaces to improve the ability of users to associate icons to meaningful objects or actions. It is possible for the user to customize icons, by cutting and pasting graphics or by drawing an icon with an icon editor. However, the process is often tedious and the result is not always informative. The only icon editing software presently available which automatically enhances the informative nature of an icon are the programs which create thumbnail graphic icons for graphic files. With these programs, a file which contains a picture of an elephant, for example, will be provided with an icon which is a miniature version of the elephant picture. Since these programs do not apply to files made up of text or for executable program, they do not provide a general solution to the problem of indistinguishable graphic icons.
Even for software application developers it is becoming increasingly difficult to design meaningful graphical icons that satisfy the constraints imposed by existing WIMP interfaces and that are different from those already in use. One approach to the problem of designing meaningful graphics for icons has been to work with animated and multidimensional icons. It is believed that animations can be used to improve the expressiveness and extend the amount of information that can be conveyed in an icon. Some of this research has been incorporated into existing operating system interfaces, particularly for generic process depiction. For example, when an operation on a file is performed or a program is opened, the mouse pointer may become momentarily animated or may assume a different graphic, e.g. by displaying an hourglass. However, there are serious limitations on the use of animated icons in current operating systems interfaces. First, they are only possible for cursor animations. It is not currently possible, even for application developers, to supply animated icons for file objects because existing operating system interfaces do not provide support for such icons. Second, cursor animations are constructed by operating system developers and fixed in the operating system. Software developers can make use of alternative cursor animations but they must select a cursor animation from the small set of choices that are included with the operating system. The set of animated icons is fixed and finite.
Another regimented aspect of the current graphical interfaces is that they are relatively static. With a few exceptions, such as the animated cursors described above or the zooming open of windows, the graphical display is inanimate. While the operating system interface presents a static graphical representation of objects such as files and directories, there is not much graphical representation of processes performed by the operating system. Thus, as the user initiates a process (such as copying a file or launching a program, etc.) there is no intuitive indication by the operating system to the user as to what is happening. For example, the hourglass animation of the mouse cursor may indicate that the operating system or program is performing some function but there is no indication of what that function is. Moreover, even animations such as the hourglass or zooming of windows that are indicative of processes, cannot be used for graphical display of interactions with the representations of objects such as files and directories known to the operating system. This is another result of the fact that animations displayed by the operating system interface must be constructed in advance by software developers.
Another difficulty with WIMP interfaces for the operating system arises in the use of menus for the selection and execution of most operating system commands. For many users, this is an improvement over the old command line interfaces in which a user had to remember the correct text string identifier and avoid typing or spelling errors in order to invoke a command. However, the need to traverse menus in order to locate the correct mouse responsive text string for a command is a nuisance for many computer users. It generally requires that the mouse pointer be moved away from the workspace and that a number of hierarchically organized lists be scanned for the desired command. Although accelerator key sequences are normally available for command selection, most computer users find them difficult to learn and use. This is because they normally require that a control key be used in conjunction with another key. A user is forced to remove the hand from the mouse to press keys, an action that tends to disrupt the orientation of the mouse pointer and require recalibration of hand and eye in order to resume work with the mouse.
Recently, software developers have created application programs that allow a user to configure an alternative desktop interface to the ones provided by standard operating systems. These programs extend the underlying representation of an icon and allow icon graphics to be different sizes and shapes from the standard 32 by 32 pixel icons available in the usual operating system interface. They do this by requiring that users select icons from a large set provided by the interface developers. Edmark's KidDesk is an example of such a program that extends the desktop metaphor for use by young children. The software can be set up to provide young children with access to a small set of programs. Like windows-based software for adults, it is limited to a single graphical view, and a set of predesignated icons.
The handling of user interaction with and display of the files and directories that make up the computer's file system is a central function of any operating system interface. As noted earlier, command line interfaces which required a user to repeatedly invoke a sequence of commands like DIR and CD in order to examine the file system have been particularly difficult for users. Since it is so difficult and time consuming to navigate a file system using a command line interface, file system management programs were developed for hierarchical files systems. Most of these programs include a quasi-graphical representation of the file system "tree" so that the user can see at once (or in a few screens) how directories, subdirectories and files are organized relative to each other. File system management programs improve on the bare command line interface by continuously displaying command menus and/or file lists. The interface provided by these programs, however, is mainly text based. The user is forced to read listed information. With the exception of the actual text, all files and directories look the same, i.e. a line of text. Only the relative location of the lines of text in the hierarchical tree gives a clue as to how the files and directories are related.
WIMP interfaces for the operating system allow for improvements in the earlier file system management programs by enabling the use of separate windows for the display of directory contents and allowing some files to be executable when they are clicked on with a pointing device. In the Apple Macintosh, file system management is included as part of the operating system while Microsoft Windows and IBM's OS/2 include a similar File Manager program along with the basic operating system. In each of these systems, the user can explore and navigate through the file system by pointing and clicking on icons with the aid of a mouse or other pointing device. For example, in order to view the contents of a disk, the user would locate the mouse pointer on the icon of the disk and click the mouse button twice.
In the Macintosh, which offers the greatest functionality in file system management, the interface responds to mouse clicks by opening a window which contains icons representing directories and files contained on the disk. Beneath, or alongside, each icon is the name of the file or directory. When displayed in one mode, each icon resides on a line followed by the name of the file, the size of the file, the date it was modified, etc. By simply pointing and clicking the mouse, the user can rearrange the icon display alphabetically, chronologically, by size, etc. The icons remain visible on the screen until the user closes the window with an appropriate mouse click. If there are more icons than can be displayed on the screen, the window contents can be scrolled horizontally and vertically. This is much more useful than the directory list in a command line interface which scrolls quickly off the screen and cannot be scrolled backwards. Moreover, each of the directory icons will appear to respond to mouse clicks by displaying their contents either in another window, or in a hierarchical tree within the same window. Depending on the size of the display screen, the user may view the contents of several directories side by side. Files and directories can be moved or copied by clicking on their icons and dragging them onto the icon of the directory or disk to which they are to be copied or moved. This is much more convenient than typing "copy directory1.backslash.subdirectory1.backslash.filename directory2.backslash.subdirectory2.backslash.filename" to copy a file. Moreover, several icons can be selected by the mouse and dragged as a group to a new location. Files, groups of files, and entire directories are deleted by dragging them to a graphic icon that depicts a trash can. Files and/or groups of files can be opened, or programs executed, by clicking with the mouse. Some program icons may be responsive to "drag and drop" operations so that if a file icon is dropped onto the program icon, the program will perform some operation on the file.
Improvements in usability of WIMP based interfaces to the file system are mainly a result of the fact that better use is made of the computer monitor "real estate" to organize and display information about the relationships among directories and files. With these interfaces, computer users are able to view the file system structure in a few screens or windows. When necessary, the use of a pointing device makes it easy to switch among these windows to refresh one's memory and develop a complete mental picture of the file system structure. Because the capacity of storage devices such as hard disks and CD-ROMs is increasing and networked file systems are becoming prevalent, existing interfaces for file management are not able to effectively aid users attempting to manage or browse the enormous numbers of files now available to them. Very large numbers of windows must be opened to traverse a large file system and displays of directory trees have begun to require many screenfuls of text. When this is the case, graphical displays of the file system begin to resemble the old command line interfaces because it is no longer possible for a user to examine the file system structure in a small number of views.
There has recently been a great deal of research focused on improving the ability of users to organize, browse, and retrieve files from very large file systems. Advances in computer processing power and computer graphics have enabled the development of software tools that attempt to utilize the capacity of the human visual system for rapid processing of large volumes of information. Views, visual abstractions, and other information visualization techniques have been applied to the problem of finding and organizing files in a computer file system. For example, Cone Trees increase the amount of information (e.g. the number of files displayed) by extending diagrams similar to those provided by existing file management programs into three dimensions and adding animation (e.g. rotation of the trees). These techniques are based on the use of overviews and visual abstraction of directory structure. They may be useful for navigating a file system structure in which the files are either already known to the user or are easily describable by text string names. They do not offer much help to a user exploring unknown file systems such as would be found on a network because text string names are generally inadequate as descriptions of file or directory contents.
Abstraction oriented methods work by removing cues that are not directly relevant (e.g. by displaying only the structure of a file system). For these tools, operating system limitations on graphic icons described earlier are not a problem because small generic icons and/or text strings are the preferred representation for file system objects. A different approach to the problem users have locating and identifying files in a computer file system is to support the human proficiency in using unusual features of phenomenon to index and retrieve information. MEMOIRS, a file management tool designed for adults, uses this approach by providing facilities to trigger memory for events as cues in recognizing and indexing files. However, event based retrieval cues are even less useful than text string names when browsing an unfamiliar file system. None of these methods has made use of the known human capacity for making use of detailed visual information and the use of distinguishing visual detail for both recognition and recall. Presently exploration of unfamiliar file systems, learning file system structure, and retrieval of particular information from a file system must take place with few effective mnemonic cues. Moreover, it is not possible to utilize the power of complex, detailed images to convey information in order to orient and acquaint a user with the contents of an unfamiliar file system. While "a picture is worth a thousand words", explorers in cyberspace must contend with a two or three word description and, perhaps, a 32 by 32 pixel icon to indicate the contents of a directory or a file.
Existing interfaces for file management, like that of the Macintosh, have been designed in accordance with the desktop metaphor. The use of windows to demarcate different directory listings, text string descriptions of files and directories, and even the graphic icons of file folders and paper pages that denote directories and files have been constructed to resemble a desktop environment. While the desktop metaphor works well for task switching among applications, and the windows metaphor is suitable for applications in which text or numeric symbols are organized into separate documents, e.g. text processing or spreadsheets, for tasks in which traversal of a space is the predominant characteristic a "worlds" metaphor is more appropriate. Here the space can be organized in a graphical analogue to the physical world in which magical and physical properties can be intermixed to fit the task (Smith, 1987). One can move through the abstract space by physically traversing the graphical world represented on the computer display. In fact, the original research on Rooms was intended to convey the impression of an complete office environment which included a desktop workspace. In its development into the Macintosh operating system interface, it's scope was restricted to the desktop alone.
Recent advancements in computer graphics outside the realm of operating systems have enabled the development of highly intuitive application programs--particularly in the areas of education and entertainment. Much of this technology has been given the epithet of "multimedia" because it combines high resolution graphics, animation, video, and sound as well as ordinary text. There are now a large number of software application programs that use multimedia to create the impression of a complex "world" that can be traversed. In these applications, the user is presented with screens of graphical information. Each screen can have several hot spots that behave like operating system icons in that they respond to pointing and clicking of the mouse pointer. Typical actions in response to clicking on a hot spot include: displaying another screen of graphics, playing a sound, displaying an animation, displaying text, displaying a video, or a combination of these actions. Navigating a well designed hypertext application can give the impression of walking around in the real world. The user can look at where the user wants to go and go there by clicking the mouse pointer on an icon that points in that direction. The user can examine objects by pointing and clicking on them. The user can pick up objects, put them away, carry them, return to where the user started and go off in another direction. Some of these applications contain virtually no text at all and the user freely "walks" through thousands of graphic screens, views video clips and hears sounds along the way. For example, the user may enter through the door of a building by clicking the mouse pointer on the door and see many bookshelves inside. As the user approaches a bookshelf, by pointing the mouse pointer at the bookshelf and clicking the mouse button, titles of the books come. into view. The user may select a book by clicking the mouse pointer on the spine of the book and the book will open showing the contents of the book. Pages are turned forward or back by clicking the pointer on the corner or edge of the page to be turned.
Hypermedia applications that make use of a worlds metaphor appear to be particularly appealing to children. The worlds metaphor has been used widely and very successfully in video games and in educational software such as Broderbund's Treehouse and Myst. Recent worlds based programs, such as Knowledge Adventure's 3D Dinosaur Adventure, use three dimensional graphics to better convey the impression of movement through a realistic space. In addition to the games programs that are now commonplace, there has been research on applications to display pictures in a museum and other types of information.
In all these cases, the use of a worlds metaphor requires that the graphical world be constructed by an application developer. With the exception of the alternative desktop programs such as KidDesk, described earlier, hypermedia software applications are not intended to be used for general access to the file system and computer operating system. Users traverse the multimedia world designed by application developers by following hypermedia links or by moving a mouse pointer through a predefined three dimensional model. A user can not add information or extend the world except in limited predefined ways. A necessary feature for an interface to the computer operating system is to provide a user with the ability to add or remove file objects in the underlying file system and their representations in the interface. Alternative desktop programs solve this problem by using hypermedia technology to allow the user to select from a fixed and finite set of graphic icons to use as representations for their files or programs. As noted earlier, a user of alternative desktop programs can only use graphics that have been predefined by the application developers, graphic icons for files and directories are independent of the context in which they reside, that is, a fixed graphic background (or user selectable set of backgrounds) is provided by the application developer, and operating system actions are not represented at all. Moreover, apart from ordinary operating systems icons, it is not possible for a software applications developer to develop a graphical representation for a program that will be visible to the user of the alternative desktop program. Instead, the user of an alternative desktop must reinstall each new application program into the desktop by linking it to one of the icons included within the desktop software. Because users cannot make use of graphical icons and backdrops designed by the application developers for their programs, icons are likely to be even less representative of file and directory content than are the limited graphic icons and text string names available in an ordinary operating system interface.
User definable hot spots that respond to mouse clicks are provided by hypermedia authoring tools such as Apple's Hypercard, IBM's Linkway and AVC, Asymetrix' Toolbook, Macromedia Director, and others. Once defined, these hotspots behave like the icons in an operating system interface in that they respond to mouse clicks by executing an action. Unlike operating system icons, hot spots defined using a hypermedia authoring tool can be represented by any graphic and can be linked to any type of behavior. Authoring tools are intended to be full featured programming languages for development of multimedia applications. In addition to allowing the definition of mouse responsive hot spots, they generally offer features including the display of raster graphic images, animation, video and sound playback, control over synchronization of sound and animation or video, and the ability to link hot spots to actions including execution of programs written in more general programming languages such as C. Many authoring tools also include draw programs for the construction of raster graphic pictures and wave editing programs for sound creation.
These programs are very powerful and give a skilled programmer the ability to create the sophisticated hypermedia applications described above such as Myst or Treehouse. Learning to use these tools to develop a hypermedia application generally takes many weeks or months and is therefore an activity normally carried out only by professionals or committed hobbyists. Moreover, it is generally necessary to make use of more specialized development tools to produce the graphics, sound, and animations required for a hypermedia application. Most applications created using these tools require programs written in more general programming languages such as C for the execution of hot spot actions or to maintain application data structures. The authoring tool simplifies the job of programming a multimedia hyperlinked application by giving a programmer ready made modules for multimedia such as animation and sound playback, and providing an interface that makes it easier to view, cut, and paste graphics and sound developed elsewhere, and to link the display of graphic scenes or execution of arbitrary actions to hotspots. Nevertheless, using these tools it normally takes many hours or months and programming by skilled artisans to develop a hypermedia application.